Determinants of grader agreement: an analysis of multiple short answer corpora

Abstract The ’short answer’ question format is a widely used tool in educational assessment, which students write one to three sentences response an open question. answers are subsequently rated by expert graders. agreement between these graders crucial for reliable analysis, both terms of strategies and developing automatic models short answer grading (SAG), active research topic NLP. This makes it important understand the properties that influence grader (such as difficulty, length, correctness). However, twin challenges towards such understanding wide range SAG corpora use (which differ along number dimensions) hierarchical structure potentially relevant can be located at corpus, answer, or levels). article uses generalized mixed effects analyze effect various on six major two main assessment tasks (language content assessment). Overall, we find broad among corpora, with behaving similarly across (e.g., shorter correct easier grade). Some show more corpus-specific behavior difficulty level), some line general tendencies than others. In sum, obtain nuanced picture how similar dissimilar from derive suggestions corpus development analysis.